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ABSTRACT 

This report synthesizes the results of activities of 
the National Center on Educational Outcomes (NCEO) focused on 
determining whether secondary data analysis of state and/or national 
data collection programs could produce policy-relevant reports on the 
status of children and youth with disabilities. The report describes 
methodology and findings of the NCEO's five-step approach: (l) 
identifying a working list of potentially useful and relevant state 
and national data sets; (2) targeting specific data collection 
programs that include indicators relevant to NCEO's conceptual model 
of educational outcomes and indicators; (3) conducting 
disability-sensitive reviews of the targeted databases; (4) analyzing 
the databases to determine the extent to which NCEO's conceptual 
model of outcomes and indicators is represented in state and national 
data collection programs; and (5) developing and completing a list of 
prioritized secondary data analyses. The report concludes that the 
exclusion of significant numbers of students with disabilities from 
both state and national data collection programs and/or the 
nonexistent or varying disabi 1 ity-specif ic variables used to identify 
these students make it all but impossible to describe the status of 
students with di sabi 1 i t i es . Recommendations are offered for improving 
the collection and reporting of information in state and national 
data collection programs. These include: develop broader and more 
uniform definitions of sample eligibility across state and/or 
national data collection programs; increase adherence to inclusion 
guidelines; and include follow-up studies and special analyses of 
ineligible students as a standard component of data collection 
programs. (Contained 30 references.) (DB) 
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Abstract 



Education reform initiatives throughout the entire educational system have focused attention on 
outcomes and quantifiable data- With increasing frequency, the data needed to monitor and 
evaluate reform initiatives are being drawn from large-scale state and national data collection 
programs. Although sufficient national level school completion outcome indicators and state goal 
statements suggest the potential for evaluating the outcomes of students with disabilities, secondary 
da ta analysis of state and/or national data bases is not feasible. The exclusion of significant 
numbers of students with disabilities from both state and national data collection programs and/or 
the variable or nonexistent disability-specific variables used to identify these students in these 
programs make s it all but impossible to describe the status of students with disabilities. 
Recommendations are offered for improving the collection and reporting of information in state and 

national data collection programs. 
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Why We Can't Say Much About the Status of Students 
with Disabilities During Education Reform 



"School reform has riveted national attention on the numbers" (Hanford & White, 1991). 
Reform «»to educational systtmhavc 

and quantifiable data. With increasing frequency, the data needed to morntor and evaluate these 
reforms are being drawn from large-scale national data collection programs (e.g.. National 
AsseTmmof&lSnal PtogSss - NAEP) and statewide assessments (e.g„ state-wtde tesung). 



attainment The Special Study Panel on Educational Indicators (1991) reported that thesu< 
Somd reformdepends on the development of a "comprehensive educatt^indica^ws 
information system capable of monitoring the health of the enterprise, identifying problems, and 
iUuminating the road ahead" (p. 6). Similar calls for the improvement of the !!? W been 
education data svstem and the development of new components to mclude in the system have been 
echoed in reports by the National Education Goals Panel (1991) and the 

Statistics Agenda Committee of the National Forum on Education Statistics * ese 

Piouns has turned toward national data collection programs for indicators to monitor progress 
Hgte ££5 ^TJSkm. Parallel activities LvealsooccoKed iMtaj « as they have 
grappled with documenting the effects of state and local education reform initiatives. 

Even with the past year of upheaval and questioning abom continuing Go^ 200°, few^^ 
have challenged the need to collect data on academic outcomes. There is a commim^nt to the need 
for data on the effects of education for students, particularly students with disabilities. 

In addition to the general education reform movement, reform initiatives in S P^1 
education (Skrtic, 1991) have produced increased interest in the analysis of existing national and 
Sate databases Since the passage of PL 94-142 in 1975, there has been over a decade of 
SSSes that have focused primarily on the issue of educational access for students with 
disabilities and implementation of the processes embodied in the law. Increasingly the question of 
"where’s the beef?" has been asked from both within and outside of special 
recently turned toward evaluating the outcomes of special education, or, where s the data? on 
effectiveness (DeStefano & Wagner, 1991). 

Purpose of this Report 

The National Center on Educational Outcomes (NCEO) was established in October, *990 
to work with state departments of education, national policy-making groups, and others to facilitate 
development and use of indicators of educational outcomes for scents with 
disabilities. It is believed that responsible use of such indicators will enable students with 
disabilities to achieve better results from their educational exigences. Students wtihawide array 
of disabilities fit within this population, including those with learning disabilities, emo0 ^ . 
disabilities and speech and language impairments, those with sensory disabilities such as hearrng 
^pSnisVdS ZpdEA. and those with multiple and more severe disabilities, typically 

involving significant mental impairments. 

One of the four major strategic goals of NCEO during the Past fiw to h^ "» 

enhance the availability and use of outcomes information m decision malang at the federal and state 
level " Several activities have been subsumed under this goal. Two activities have dealt 
^cajfr^th Bftdnj and reviewing the characteristics of state and national data sets as a 
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prelude to secondary analysis of the data. The purpose of these activities has focused on 
determining the feasibility of extracting policy-relevant information on the educational status and 
performance of students with disabilities. 

The results of this critical review process have contributed to NCEO’s strategic goals by: 

Determining the extent to which the current state and national data collection 
programs consider students with disabilities when planning and 
implementing data collection. 

Determining the extent to which students with disabilities are included or 
excluded in state and national data collection program*. as a result of sample 
design and inclusion/exclusion procedures. 

Determining the depth and breadth of outcomes included in state and 
national data collection programs and the extent to which these outcomes 
approximate a conceptual model for a comprehensive system of indicators. 

Determining the means by which state and national data collection programs 
describe the functional characteristics of students with disabilities. 



During the past five years, NCEO staff have systematically examined the extent to which 
students with disabilities are "at the table" when it comes to participation in state and national data 
collection activities. Given that nearly 5 million school-age youngsters with disabilities receive 
some form of special education services, services that are provided at significant expense to our 
educational system, it is imperative that we examine how these students are performing. This 
report synthesizes the results of the NCEO activities that focused on determining whether 
secondary data analysis of state and/or national data collection programs could produce 
policy-relevant reports on the status of children and youth with disabilities (McGrew, Spiegel, 
Thurlow, Ysseldyke, Bruininks, & Shriner, 1992). 



The NCEO Approach 

The NCEO approach to secondary dam analysis of state and national data collection 
programs followed a five step plan. The five steps were: (1) identify a working list of potentially 
useful and relevant state and national data sets, (2) target specific data collection programs that 
include indicators relevant to NCEO’s conceptual model of educational outcomes and indicators, or 
that are featured prominently in the national educational indicator dialogue, (3) conduct disability 
sensitivity reviews of the targeted databases, (4) analyze the target databases to determine the extent 
to which NCEO’s conceptual model of outcomes and indicators are represented in state and 
national data collection programs, and (5) develop and complete a list of prioritized secondary data 
analyses. An outline of this process is presented in Figure 1. 

This report summarizes the results and experiences of NCEO in following its five-step plan 
for producing reports on the status and performance on children and youth with disabilities. 



Identification of a Working List of Databases 

The identification of a preliminary list of national and state databases required two different 
sets of activities. 
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Figure 1. NCEO Approach to Identifying and Conducting Secondary Data Analysis of State and 
National Databases 
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National data collection programs. The process of locating national data programs required 
the use of a mixture of formal and informal sources. Written, phone, and personal contacts were 
made by NCEO staff with a wide array of public and private agencies and organizations devoted to 
either the production (e.g., National Center for Education Statistics - NCES; National .Center for 
Health Statistics - NCHS) or dissemination (e.g., National Technical Informatio n Service - NTIS; 
Data Users Services Division of the Census Bureau) of database information. Policy oriented 
reports produced by organizations that utilized secondary analysis of data were reviewed for 
possible clues about additional data sets. In addition, critical reports by groups actively involved in 
recommending the use of data from the current national education data system (National Education 
Goals Panel; Special Study Panel on Educational Indicators; National Education Statistics Agenda 
Committee) were reviewed for references to important data sets. Articles devoted to the listing of 
data sets (e.g., Taeuber & Rockwell, 1982, "National social data series: A compendium of brief 
descriptions") were also found to be useful in the identification of additional data sets. Finally, 
infor mal sources were also used and consisted of referrals to sources from individuals contacted at 
various agencies or organizations and contacts with researchers who are visibly active in the 
utilization of data from the national education data system for policy relevant research. This 
process resulted in the development of a "working list" of over 70 national databases (McGrew et 
al., 1992). 

State data collection programs : In the Spring of 1991 , state directors of special education 
or their designees responded to the annual NCEO national survey of state special education 
outcomes activities (NCEO, 1992). This survey was used to gather information on state efforts in 
the areas of federally-reported data, assessment of outcomes, inclusion of students with disabilities 
in state assessments, state assessment needs and highlights, activities in selected outcome areas, 
and practices, programs, and plans related to outcomes. 

In the initial annual survey, 49 of the 50 states reported that some students with disabilities 
took part in their general education large-scale achievement assessments. These state assessments 
typically varied from the administration of nadonally-normed commercial achievement tests (e.g., 
Stanford Achievement Test) to state-developed norm-referenced or minimum competency exams. 
Slightly over half of the 50 states (n = 27; 54 %) indicated that students with disabilities could be 
identified in their data sets. In other words, some variable was present in the state database that 
indicated each student's special education status. These 27 suites were the initial working list 
selected for inclusion in NCEO’s secondary data analysis activities. 



Identification of Target Databases 

Die process of targeting specific state and national data collection programs proceeded in 
the following manner: 

National data collection programs . A major NCEO activity has been the development of a 
conceptual model of educational outcomes for children and youth with disabilities (Ysseldyke, 
Thurlow, Bruininks, Gilman, Deno, McGrew, & Shriner, 1992). As a result of this model devel- 
opment process, the major outcome domains were identified for the assessment of outcomes for all 
students at six key developmental points: Early Childhood- Age 3 (Ysseldyke, Thurlow, & Gilman, 
1993a); Early Childhood-Age 6 (Ysseldyke, Thurlow, & Gilman, 1993b); Grade 4 (Ysseldyke, 
Thurlow, & Erickson, 1994a); Grade 8 (Ysseldyke, Thurlow, & Erickson, 1994b); School 
Completion (Ysseldyke, Thurlow, & Gilman, 1993d); and Post-School (Ysseldyke, Thurlow, & 
Gilman, 1993c). These were identified by key stakeholders (i.e., policy makers, teachers, par- 
ents, legislators, advocates, etc.) who participated in a consensus meeting process (Vanderwood, 
Ysseldyke, & Thurlow, 1993). The school completion version (Ysseldyke, Thurlow, & Gilman, 
1993d) of this dynamic model is presented in Figure 2. National data collection programs that 
appeared to include indicators of the NCEO outcome domains were identified. 
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Fi gure 2 . NCEO Conceptual Model of Education Outcomes 
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A second consideration focused on national education reform initiatives. The reports of the 
major national groups (e.g., National Education Goals Panel; National Education Statistics Agency 
Committee of the National Forum on Education Statistics; Special Study Panel on Educational 
Indicators) that were either (1) developing comprehensive systems of indicators, or (2) making 
recommendations on how to improve the national education data system were reviewed to identify 
national data collection programs that were receiving significant attention (McGrew et al., 1992). 
Based primarily on these two considerations, 28 national data collection programs were targeted. 
The list of targeted national data sets is presented in Table 1. 

State data collection programs . Individual follow-up phone calls were made by NCEO 
staff to the 27 states that were identified (as a result of the NCEO state survey) as ones that might 
have usable data sets for secondary data analysis. Potential data sets were defined as those that 
included achievement data from large-scale achievement testing programs. 

After a lengthy process (see description by McGrew, Spiegel, Thurlow, Shriner, & 
Ysseldyke, 1994 for description), only 6 of the 27 targeted state databases were acquired by 
NCEO staff. Seven general types of problems were encountered when trying to obtain the targeted 
data sets from 21 of the states. These problems included nonresponses to requests for data, 
concerns about confidentiality, computer files with suspect or unreadable data, excessive 
acquisition costs, and unreliable identification of students with disabilities in the databases. 



Disability Sensitivity Reviews of Targeted Databases 

Contacts were made with the appropriate state or national agency staff to request additional 
information on the targeted data collection programs. Requests were made for all relevant 
methodology and technical reports and manuals. For the state data, formal requests were made to 
secure copies of the computer files. All relevant materials for each targeted data set were subjected 
to a detailed review process (see description by McGrew et al., 1994) that focused on determining 
the extent to which each database might be suitable for secondary data analyses. Specific attention 
was focused on determining the extent to which students with disabilities were included and 
appropriately identified in die final database. 

Inclusion of students with disabilities in national and state data collection programs . A 
review of state and national data collection programs found that "the ability to extract useful policy- 
relevant information on the outcomes of students with disabilities is hampered by the significant 
exclusion of portions of this population in a number of these data collection programs" (McGrew, 
Thurlow, & Spiegel, 1993, p. 327). NCEO found that most existing state and national data 
collection programs excluded a large portion of the student population with disabilities. At the 
national level, 40% to 50% of school-age students with disabilities were estimated to be excluded 
from the most prominent national education data collection programs (e.g.. National Assessment of 
Educational Progress; NAEP). Conversely, exclusion was found to be minimal in a number of 
noneducational national data collection programs (e.g.. National Health Interview Survey). 
Indicative of the low status placed on the inclusion of students with disabilities in state assessment 
programs was the finding that data documenting the extent of exclusion was, with few exceptions, 
largely unavailable (McGrew et al., 1992). 

Although available assessment technology precludes the inclusion of all students with 
disabilities in large scale assessment programs, NCEO concluded that a "sizable portion of 
excluded students should not have been excluded and could readily participate (some with testing 
accommodations; others without)" (McGrew, Thurlow, & Spiegel, 1993, p. 347). 



NCEO Synthesis Report 



Table 1 

NCEO Targeted National Data Collection Programs 



Department of Education . 

Transcript Studies, National Adult Literacy Survey, National Assessment of Educational Progress, 
National Assessment of Educational Progress: Trial State Assessment, National Education 
Longitudinal Study, National Longitudinal Transition Study of Special Education Students, 
National Household Education Survey, Young Adult Literacy Survey, Beginning Postsecondary 
Student Longitudinal Study, Baccalaureate and Beyond, Early Childhood Longitudinal Study 

Department of Health and Human Services . 

National Health Interview Survey, National Health and Nutrition Examination Survey, National 
Survey of Family Growth, National Adolescent Student Health Survey, Youth Risk Behavior 
Surveillance System, National Household Survey of Drug Abuse, Monitoring the Future 

Department of Commerce 

Current Population Survey, Survey of Income and Program Participation 
Department of Labor 

Workforce Participation Survey, Workplace Literacy Assessment 

Department of Justice 
National Crime Survey 

National Science Foundation . 

Survey of Graduate Students and Post Doctorates, Longitudinal Study of American Youth 

American Council of Education 
General Education Development Testing 

The College Board 
Advanced Placement Tests 



Many factors have been found to contribute to the exclusion of students with disabilities. 

Among these factors are: 

• Use of vague assessment inclusion/exclusion guidelines 

• Differential and inconsistent implementation of inclusion/exclusion guidelines 

• Incomplete or unsuccessful monitoring of the extent to which the intent of 
inclusion/exclusion guidelines is followed 

• Sampling plans that systematically exclude students who are in separate schools 
and students who are not in graded programs 

• Altruistic motivations, such as lessening the emotional distress to students who 
are not expected to do well 

• Nonavailability of accommodations in assessment materials and procedures 

• Incentives created by the desire to have a school or state look good in 
comparison to others in the state or nation 
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Identification of students with disabiliti es in national and stats data collection programs - 
Although large numbers of students with disabilities are excluded from many state and national 
assessment programs, some students with disabilities (almost all in the case of many 
noneducational national data collection programs such as those of the National Center on Health 
Statistics - NCHS) are included in these programs. Thus, NCEO considered it important to 
investigate the extent to which subpopulations of students with disabilities were identified m the 
final database, when and if they were included in data collection programs,. Accurate and 
consistent identification of students with disabilities in state and national databases is a prerequisite 
for conducting useful secondary data analysis. 

Nineteen of the twenty-eight targeted national data collection programs were reviewed to 
evaluate the correspondence between the disability categories or terms used in each data collection 
program and those used by the federal government in its Annual Report to Congress QILths 
Implementation of the Individuals with Disabilities Act (McGrew, Algozzine, Spiegel, Thurlow, & 
Yssledyke, 1993). The results indicated that there is significant variability between different 
national data collection programs in the manner in which individuals with disabilities are identified. 
Notable differences were observed between data collection programs sponsored by educational and 
noneducational agencies. As would be expected when using the federal special education 
categorical system as the basis for the analytical framework, those data collection programs 
sponsored by the U. S. Department of Education contained the largest number of data elements that 
displayed some correspondence to the federal special education categories. However, even withm 
data collection programs sponsored by the Department of Education there was variability. 

Inconsistency in the identification of students with disabilities in state databases was also 
found to be a problem. Of the initial 27 states contacted, 6 state databases were characterized by 
unreliable or no coding of students with disabilities. In fact, few states provided for the 
identification of students with each type of disability at the grade levels assessed. Further 
complicating the ability to aggregate state information was incompatibility across states m the 
differentiation within categories (e.g., mental retardation) by level of disability (e.g., educable, 
trainable, or severe versus a single global category). In addition to problems in the exclusion of 
many students with disabilities, and inconsistent or no coding of students with disabilities in state 
data collection programs, aggregation of state databases was also problematic due to the sparseness 
of data at individual grade levels (even after aggregation) and noncomparability of types of data 
(national vs. local norm-referenced scores; state-specific minimum competency scales). 

• 

Based on the extent of exclusion and variable or no identification of students with 
disabilities in most state and national data collection programs, we concluded that it is currently not 
possible to extract, on an regular basis, nationally representative policy-relevant information on the 
educational and quality-of-life outcomes for students with disabilities. This conclusion echoes the 
conclusion reached by the Committee on a National Agenda for the Prevention of Disabilte (Pope 
& Tarlov, 1991) that epidemiologists and statisticians have generally ignored the issue of disability 
and that the information that is available has been collected in a piecemeal fashion. 




*s Comprehensive System of Out comes and 



Indicators . 



A goal of NCEO is to evaluate the extent to which the indicators included in the NCEO 
comprehensive system of outcomes and indicators (see Figure 2) are present in existing and 
emerging state and national data collection systems. The value of this activity is to: (a) highlight 
potential gaps in current state and national education data collection systems, systems that have not 
had the benefit of evolving from an a priori conceptual model, and (b) provide an organizational 
framework from which to conduct secondary analyses. The identification of information gaps in 
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current data collection systems may serve to stimulate the modification of current data collection 
programs, or, the development of new data collection programs that produce more comprehensive 
conceptually-based data that provide useful, policy-relevant information for all children. 

Analysis of national data collection programs, _T he analysis of targeted national data sets 
followed a sequence that included the review and classification of individual test or survey items 
according to the NCEO conceptual model of outcomes and indicators (see details m McGrew, 
Spiegel, Thurlow, & Kim, 1994). To date these analyses have been completed for the NCEO 
school completion model, which includes 77 indicators. Thirteen national data collection programs 
that measure relevant outcomes at the completion of school were reviewed. 

At the national level, indicators of important school completion outcomes for all students 
were found to be substantially represented in the existing national data collection system. Across 
13 national data collection programs that assess individuals at the completion of their secondary 
schooling, 91% (70 of 77) of the NCEO outcome indicators were found to be represented by at 
least one measure. Approximately half of all the NCEO indicators were found to be present m two 
or more of the thirteen data collection programs reviewed. Although the final number of usable 
indicators would most likely be less, due to the technical and methodological problems typically 
encountered in secondary data analysis (McGrew et al., 1991), McGrew, Spiegel, Thurlow and 
Kim ( 1994) concluded that the potential exists for producing comprehensive and conceptually 
organized policy reports regarding the status of all students as they complete school on a number ot 
important NCEO outcomes and indicators. 

The NCEO review indicated that the outcome domains of Satisfaction and Accommodation 
and Adaptation, and to a smaller extent, Personal and Social Adjustment and Responsibility and 
Independence, are the areas where usable national indicators may be problematic. Outcome 
domains for which usable national indicators are most likely to be found are Participation, 

Academic and Functional Literacy, Contribution and Citizenship, and Physical Health. 

State data collection programs. Despite a high degree of overlap between manystate 
education goals and the outcome domains in the NCEO conceptual model (Span de & Thurlow, 
1994), Ysseldyke, Vanderwood, and Reschly (1994) found that there are few NCEO school 
completion indicators for which data are currently reported by seven or more states. For state data 
collection programs, data managers from 36 states and territories were surveyed on the availability 
of data for the 77 NCEO school completion indicators. The state data managers were asked to 
indicate on a 5-point Likert scale the extent to which the data for each indicator are availableor 
might be made available. In general, state data are currently available for indicators in the Presence 

and Participation and Contribution and Citizenship domains. Data on Physical Health, 
Independence and Responsibility, Satisfaction, Accommodation and Adaptation, and Personal and 
Social Adjustment are relatively unavailable across states. These results suggest a significant 
disparity between the kinds of data stakeholders say are needed for purposes of lmprovuig 
instruction and educational outcomes and the data that are currently available or could be made 
available. 

Development of a List of Prioritized S econdary Data Analyses 

The final step in the NCEO process is the generation of key informational needs and 
research questions, die placement of these questions within a conceptual framewonc, and the 
specification of the types of comparisons needed to answer the key questions (McGrew et al., 
1992). This step focuses on answering the question of "what do we want to know about students 
with disabilities?" Unfortunately, given the nature of the problems encounte^ when^emptmg to 
locate usable state and national level data on the status of students with disabilities, NCEO has as 
yet been unable to reach this step. Instead, NCEO has focused its energies on helping state and 
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national agencies find solutions to the challenges of increased inclusion, better sample description, 
and increased participation through the development and use of assessment accommodations and 
adaptations for students with disabilities. It is only after progress is made in these areas, and 
"good" data become available in state and national data sets, that serious attention can be directed to 
the development and completion of lists of secondary analyses. 



Summary and Conclusions 

Calls for reform in American education during the past decade have resulted in raised 
expectations, attempts to develop "world class" standards, and increased interest in the 
measurement of school outcomes. There is no doubt that the current wave of education reform 
places significant attention on measurement-driven accountability and evaluation. Results from 
state and national assessments and national reports (e.g„ annual National Education Goals Report) 
often appear in local and national newspapers in the form of eye-catching headlines such as "eighth 
grade students in state X rank 20th in the nation in mathematics" or "X percent of high school 
students experience violence during school." 

A look behind the headlines reveals that the data such reports are based on are derived 
primarily from large-scale state and national data collection programs. Further examination 
indicates that an answer to the question of "How are students with disabilities doing in the same 
areas?" is currently impossible to answer. Given that the current education reform activities use 
measurable indicators from large-scale assessments as the index of progress, the evaluation of the 
education of most students with disabilities is being short-changed. 

The potential exists for the production of useful policy-relevant information. about the 
school completion outcomes of students with disabilities. A significant number of important 
school completion outcome indicators identified by key stakeholders (the consensus-based NCEO 
model of outcomes) is present in national data collection programs, and many are mentioned as 
important outcomes in state goals reports. The bad news is that this potential cannot ctmently be 
tapped for the recurring analysis and generation of reports about students with disabilities. The 
exclusion of significant numbers of students with disabilities from both state and national data 
collection programs and/or the inconsistent or nonexistent disability-specific variables used to 
identify students in these data collection programs makes it all but impossible to use all the outcome 
information that is currently available. In addition, although state goals encompass many important 
school completion outcome indicators, available state data collection programs only assess a limited 
number of these indicators. 

The conclusions reached in this report should not be construed as a general indictment of 
most state and national assessment activities. It is important to recognize that the problems 
encountered in NCEO’s secondary data analysis activities surfaced when attempting to use 
databases ori ginall y developed for different purposes. Most large-scale state and national data 
collection programs provide extremely important, reliable, and valid information for general 
education state and national level analyses and decision making. Many of the national programs 
were not originally designed to provide answers to educationally related questions (e.g., data 
collection programs of the National Center for Health Statistics) and are burdened with many 
competing goals and objectives. Large-scale state and national data collection programs are 
designed and operated to meet the unique needs of each state or congressional mandates. They 
typically are not designed to meet the needs of independent researchers who seek to conduct 
secondary data analyses, especially aggregated analyses across a number of data programs. 

However, in an environment of limited resources for new and expensive large-scale 
nationally representative data collection programs, we believe it would be most practical and cost- 
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effective to modify and use existing data collection programs to answer new and emerging 
questions. How individuals with disabilities are performing during and after their educational 
careers is an important public policy and research questio i that needs to be addressed now and in 
the future (Ysseldyke, Thurlow, Bruininks, Gilman, Deno, McGrew, & Shriner, 1992). 

Although specially designed national studies focused exclusively on the population of students 
with disabilities (i.e.. National Longitudinal Transition Study of Special Education Student s ) 
provide valuable information, such special surveys are expensive, limited in coverage of outcome 
domains, provide no comparable data on students without disabilities, and are typically fixed 
duration studies that do not provide information as part of a recurring national information system. 

Finally, "given the magnitude of federal and state support for educational programs for 
students with disabilities, support that reflects the valuing of this population in our society, it is 
time that this implied value be matched by the commitment of resources to address the numerous 
political and technical hurdles that must be overcome in order to be able to extract useful and ^ 
routine information on the educational and quality of life outcomes for individuals with disabilities 
(McGrew, Algozzine, Spiegel, Thurlow, & Ysseldyke, 1993, p.l 1). Although currently it is not 
possible to produce routine, quality information regarding the educational outcomes of students 
with disabilities through the secondary analysis of data gathered through large-scale state and 
national data collection programs, this does not mean this approach should be discarded. Toward 
the goal of improving the collection and reporting of information from the analysis of large-scale 
state and national data collection programs, NCEO has made a number of suggestions and 
recommendations. 



Recommendations 

1 . The most important recommendation is to increase the inclusion and participation of 
students with disabilities in state and national data collection programs. This can be done by 
adopting a number of strategies that are described in greater detail by McGrew, Thurlow, and 
Spiegel (1993), Ysseldyke, Thurlow, McGrew, and Vanderwood (1994) and Ysseldyke, 
Thurlow, McGrew, and Shriner (1994). 

a. Develop broader and more uniform definitions of sample eligibility across state 
and/or national data collection programs. 

b. Increase adherence to inclusion guidelines. 

c . Develop state and national assessment sampling frames that are more inclusive 
(e.g., do not exclude students in ungraded programs or those residing in separate 
facilities) 

d. Include follow-up studies and special analyses of ineligible students as a standard 
component of data collection programs in order to accurately estimate the effect of 
exclusion on important statistical estimates and to allow reentry into the sample 
when a data collection program is longitudinal. 

e . Increase partial participation in data collection programs, such as data collection 
components that do not require direct student responding (e.g., record reviews, 
third-party informant ratings, etc.). 

f . Include students with disabilities during instrument development to identify items, 
questions, tasks, or procedures that may need to be eliminated or modified in order 
to allow more students with disabilities to participate. 
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g. Develop appropriate assessment modifications, accommodations and alternatives 
for students with disabilities. Also, initiate research and development activities that 
focus on investigating the extent to which modifications and/or accommodations 
can be made to existing instruments used in large-scale data collection programs 
without destroying the basic psychometric properties of the instruments. 

2. Expand recurring state and national educational data collection programs to include 
other outcome domains besides academic achievement. Important outcome information in the 
NCEO conceptual model domains of Personal and Social Adjustment, Responsibility and 
Independence, Physical Health, Contribution and Citizenship, and Satisfaction would provide a 
more comprehensive picture of the status of all children. More importantly, assessments in many 
of these non-achievement domains would not have to be paper-and-pencil "tests," but could be 
gathered through other methods such as administrative record reviews and third-party informants 
(e.g., parent and teacher surveys). 

3. Initiate a dialogue among appropriate state assessment personnel (e.g., state data 
managers) on the feasibility of using a common set of data gathering and reporting strategies, 
guidelines, and/or standards that might produce more common or related data elements specific to 
students with disabilities across state assessment programs. Cooperative efforts similar to those 
that produced the Standards for Education Data Collecti on and Reporting (SEDCAR) (CEDCAR, 
1991) might be particularly worthwhile. 

4. Individuals and organizations developing instrumentation for state and national data 
collection programs should review the instruments and methodology reports of the National 
Longitudinal Transition Study of Special Education Students (NLTS) for ideas on how to develop 
new indicators or strategies for measuring important outcomes for all students. 

5 . A more uniform and standard disability variable system that parallels the federal special 
education categories should be used for educationally oriented state and national data collection 
programs, particularly those sponsored by the U. S. Department of Education. Such a system 
should be used not only to identify those individuals with disabilities who are excluded from these 
data collection programs, but also to identify those individuals who do participate. 

6. A dialogue should be initiated between representatives from the appropriate educational 
(e.g., NCES, NCEO, OSEP) and noneducational (e.g., NCHS, Census Bureau, etc.) federal 
groups and agencies to identify means by which uniform disability-related variables could be 
collected across agencies, particularly for the school-age portion of each data collection program. 
The feasibility of using the same special education categorical variable system recommended for the 
U. S. Department of Education should be examined. Alternatively, the feasibility of developing 
"cross-walk” procedures that would allow the different disability information collected by different 
agencies to be converted to the federal special education categories should be explored. 

7 . Much has been written about the significant variability between states in the 
operationalization of the same federal disability categories, and the variability between 
professionals when implementing the same operational criteria (Ysseldyke, 1987). These sources 
of variability can introduce unknown sources of error into any national statistical estimates that 
might be reported by different disability categories. Although accurately describing the disability 
population as identified by current practice, research is needed to determine whether the 
classification of individuals included in the samples of state and national data collection programs 
can be made more uniform. Research and development activities are needed to explore the 
advantages and disadvantages of developing a small set of standard personal competency variables 
(e.g., academic, cognitive, adaptive, social, emotional, physical) that could be used in all data 
collection programs, the results of which then could be used to describe and operationally classify 






NCEO Synthesis Report 



the sample respondents according to the federal special education categories. 

8 . Individuals charged with the design of instruments used in state and national data 
collection programs should consider including additional variables that would better describe those 
individuals who are included or excluded. Possible new variables include a better description of a 
student’s (1) primary means of communication, (2) physical health, (3) mobility, and (4) social 
behavior. 
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